wasm: in-browser OCR via ONNX Runtime Web interop (#157 stage 1) - #158
Merged
Conversation
…stage 1) First slice of the browser ML pipeline: scanned images OCR fully client-side. docling-pdf grows an `ocr-prep` feature (pure Rust + the image crate, no ort/pdfium): the ONNX-free half of the PP-OCRv3 recognition pipeline — line segmentation, crop prep, deterministic width-batching, CTC decode, dictionary handling — moved verbatim from ocr.rs into a public ocr_prep module. ocr.rs now consumes it, so there is exactly ONE implementation of the pre/post-processing; the native path is byte-identical after the refactor (verified: the scanned-tiff conversion diffs empty against the pre-refactor output, full docling-pdf suite green). Any wasm-vs-native drift can therefore only come from the runtime's kernels — the docling-project#157 conformance story depends on that property. docling-wasm (default `ocr` feature): `ocr_image(bytes, dict, session, to)` — decode the image in Rust, segment/prep/batch via ocr_prep, delegate each width-batch to a JS `RecSession` wrapper around an ort-web InferenceSession (`run(n, h, w, data) -> {data, dims}`), CTC- decode, and export a DoclingDocument as Markdown or docling JSON. No layout model yet (stage 2): the whole image is one text region split by the ink-projection profile — right for single-column scans, degraded on complex layouts, and said so in the docs. www/ocr.html: complete drop-an-image demo — ort-web from CDN, the ~10 MB en_PP-OCRv3 model + dictionary fetched from their public hosting and browser-cached, timing readout. Compiles for wasm32-unknown-unknown and the host (rlib tests); clippy clean; the existing wasm CI check is unaffected. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
… stage 2)
Scanned PDFs and images now convert fully client-side through the real
pipeline shape: RT-DETR layout detection + PP-OCRv3 recognition via
ONNX Runtime Web, everything else in Rust — the native pipeline's own
functions, no second implementation.
docling-pdf (all under the existing ocr-prep feature, wasm-safe):
- layout.rs: input packing (640x640 CHW) and RT-DETR decoding (sigmoid
top-q, center->corners, per-label thresholds downstream) extracted as
layout_input/decode_layout; the native batch path now decodes through
the same function per page.
- ocr_prep: prep_region_lines — the exact region-crop/line-split/prep
gathering native ocr_page does — moved out and shared; ocr.rs
consumes it.
- new scanned module: the Worker::process refinement chain
(label thresholds -> resolve -> orphans -> false-picture drops ->
contained drops), page assembly with the geometric table fallback
(the lite profile: TableFormer is stage 3), and cross-page
continuation merging, re-exposed without the ml feature.
Native path byte-identical after the extraction — the scanned-tiff
conversion diffs empty against the pre-refactor output (rebuilt and
re-verified after a disk-full broke the first attempt), full
docling-pdf/docling suites green.
docling-wasm:
- ScannedConverter: feed pdf.js-rendered RGBA pages ({scale: 2} — the
native RENDER_SCALE) one by one, finish() assembles the document with
the same continuation merging as native; page_count() for progress.
- convert_scanned_image: one-shot image path (its own page at scale 1,
like native).
- LayoutSession interop mirrors RecSession: run(data) ->
{logits: {data, dims}, boxes: {data, dims}}.
- www/scan.html: complete demo — pdf.js from CDN, layout int8
(~165 MB, models-v1 release, browser-cached) + rec model + dict,
per-page progress, timing.
Stage 3 (TableFormer + enrichment in the browser) remains: the
autoregressive decoder loop is deeply ort-typed and needs its own
careful extraction pass; table regions meanwhile take the geometric
reconstruction, matching --no-table-former.
Refs docling-project#157
Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
… field The workspace-wide clippy job unifies features: docling-cli enables docling-pdf/ml alongside docling-wasm, so PdfPage grows its ml-gated `image` field and docling-wasm's struct literal stopped compiling (E0063). A cfg-aware constructor (from_cells, under ocr-prep) spells the field only when it exists; the browser pipeline builds its pages through it. Verified with the CI command verbatim (cargo clippy --workspace --all-targets -- -D warnings) plus the wasm32 target check. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
) First real-browser test fed ocr.html a dark-theme screenshot and got "(no text found)": the whole scan pipeline assumes dark ink on light paper — the line segmentation thresholds pixels *darker* than the page mean, so light-on-dark input yields garbage. New ocr_prep::normalize_polarity inverts a predominantly dark page (mean luma below mid-gray) before segmentation and inference; both browser entry points (ocr_image, ScannedConverter::add_page — layout assumes scan polarity too) run it. The native pipeline deliberately does NOT: its input is scanned paper and the conformance baseline stays pinned. Test: the inverted synthetic page misfires raw and recovers exactly after normalization; a light page passes through untouched. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The demo pages hardcoded the English PP-OCRv3 model, so any non-Latin scan (e.g. Cyrillic) decoded to mojibake through the wrong dictionary. The wasm API is language-agnostic — model and dict are injected from JS — so the fix is demo-side only: both ocr.html and scan.html get a recognition-language <select> (en / cyrillic / ch) backed by a lazy per-language ort.InferenceSession cache; models load on first use and stay browser-cached (~10 MB each). Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
SWHL/RapidOCR ships no cyrillic PP-OCRv3 export — the previous URL
404'd and ort-web choked on the HTML error page ("protobuf parsing
failed"). Point the cyrillic entry at the
cycloneboy/cyrillic_PP-OCRv3_rec_infer HF repo instead, which carries
both the ONNX export and its paired 163-char dictionary (verified:
output tensor is [n, t, 165] = blank + dict + space, matching
dict_chars' table layout).
Refs docling-project#157
Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Without the fetch-images feature (the wasm32 build) the fetch_remote stub had no caller left — fetch_cached, its only user, is feature-gated — and FsImageResolver::base_url is stored but only read by the gated remote-URL resolution. Drop the stub and allow(dead_code) the field so `wasm-pack build` stays warning-free without forking new()'s signature per feature shape. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The v3 cyrillic export reads Cyrillic but drops spaces and flips case on lookalike glyphs. Side-by-side on a rendered test line (same preprocessing as prep_line, greedy CTC): v3 gives "еш: pаs" with glued words, PaddlePaddle/cyrillic_PP-OCRv5_mobile_rec_onnx decodes the whole line exactly, spaces included. Same interop contract — input [n,3,48,W], output [n,t,852] = blank + 850 dict chars + space, matching dict_chars' table. The v5 dictionary only exists inside the repo's inference.yml, so a flattened cyrillic_v5_dict.txt ships next to the demo pages. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The paddle2onnx v5 export carries unused initializers, and ort logs a console warning for each one it prunes at session load. Harmless but noisy — cap logSeverityLevel at errors for all demo sessions. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
While a model loads or a document converts, the demos now disable the upload controls (drop zone, file input, language selector), pulse the drop-zone border and spin the status line, so a click mid-run can't start a second conversion. Model downloads stream through a fetch-with-progress helper showing a live "x / y MB" line — the 165 MB layout model no longer sits on a static message for minutes on a first visit. Switching language now warms the model immediately instead of on the next drop. Also cap ort logSeverityLevel at errors: the paddle2onnx v5 export carries unused initializers and ort logged a console warning for each one it pruned at session load. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The demo pointed at layout_heron_int8.onnx, but the release only hosts the fp32 layout_heron.onnx (172 MB) — the int8 variant is produced locally by scripts/install/quantize_models.py and was never uploaded, so the page died on a 404 before ever becoming ready. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…n't CORS-fetchable) GitHub Release assets are served from release-assets.githubusercontent.com with no Access-Control-Allow-Origin header, so a browser fetch of the layout model failed with a CORS error the moment the URL stopped 404ing. The recognition models avoid this only because Hugging Face sends CORS headers; the official docling-layout-heron-onnx on HF can't substitute — it exports a different contract (images + orig_target_sizes in, labels/boxes/scores out) than our decode_layout expects (raw logits/pred_boxes). So load the layout model from the page's own origin: prefer ./layout_heron_int8.onnx (~68 MB), fall back to ./layout_heron.onnx (~172 MB), and if neither is present open a setup note explaining how to fetch them (download_dependencies.sh) and drop one next to the page. These are the exact conformance-validated exports the native pipeline uses; *.onnx is already gitignored so the copied model isn't committed. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
….sh in www/ Point the same-origin layout paths at ./models/ and document running the installer from crates/docling-wasm/www/ — it writes ./models/ relative to the current directory, so the page reads ./models/layout_heron_int8.onnx (fallback ./models/layout_heron.onnx) with no copy step. gitignore the www/models/ tree the installer populates. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Show a running "Ns/page avg" in the status line as pages convert, log each page's wall time to the console, and print a total + average when the run finishes — so a slow multi-page PDF is measurable instead of just "page 2/10 …". Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The release now hosts layout_heron_int8.onnx, so a fresh
download_dependencies.sh drops it under ./models/ and the page already
prefers it. Report which export actually loaded ("ready (int8 layout)")
so the ~2.4x-faster path is visible instead of guessed.
Refs docling-project#157
Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…orker ORT Web ran single-threaded because SharedArrayBuffer needs the page to be cross-origin isolated, which needs COOP+COEP response headers a static file server can't set. coi.js is a service worker that re-serves every response with COOP: same-origin + COEP: credentialless (credentialless so the jsDelivr and Hugging Face cross-origin fetches still load without their own CORP header); it reloads once on first visit to take control. With isolation in place both demos set ort.env.wasm.numThreads to the core count (capped at 8); browsers without support stay single-threaded and still work. SIMD is auto-selected by the ORT bundle. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Rasterize page p+1 while page p is being converted. pdf.js raster is main-thread and the ORT layout/OCR inference runs on worker threads (once cross-origin isolated), so the render of the next page now overlaps the convert of the current one instead of running strictly after it. getImageData hands back an independent buffer, so a single canvas ping-pongs across pages safely. Per-page timing now reports convert time plus an honest wall-clock total that reflects the overlap. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Even with multi-threaded ORT, the main thread still froze during a conversion: pdf.js rasterizes on it, and the Rust pre/post-processing inside add_page (line segmentation, CTC decode, reading-order assembly) runs synchronously on the calling thread. Move the whole pipeline off the main thread so the page stays interactive. Factor the DOM-free pipeline (model loading, rasterization, OCR/layout) into pipeline.js — parameterized only by an onStatus reporter and a makeCanvas factory — and drive it from worker.js, which rasterizes into an OffscreenCanvas and posts status/result back. scan.html becomes a thin UI: it spawns the worker, relays progress to the status line, and transfers the file bytes in. If module workers or OffscreenCanvas are unavailable it falls back to running pipeline.js on the main thread (functionally identical, just blocking); a worker-side convert failure retries once on the main thread so a result is always produced. Also folds in an ORT warm-up (one blank layout+rec inference at boot) so the first real page no longer pays the lazy kernel/thread init. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Running pdf.js inside the Web Worker fell back to its "fake worker" mode (a worker can't reliably spawn pdf.js's nested worker), which garbled the rasterized line crops: layout still found the page structure (it downsizes the whole page to 640x640) but recognition decoded noise. Split the two: pdf.js rasterization runs on the main thread again — a real HTMLCanvas, correct pixels, pdf.js's own worker — and only the heavy wasm work (layout + OCR inference and the synchronous Rust pre/post-processing in add_page, which is what actually froze the UI) runs in the worker. Ready RGBA buffers transfer to the worker per page; rasterizing page p+1 overlaps converting page p. The worker no longer imports pdf.js and needs no OffscreenCanvas, so the only requirement for the fast path is module Worker support; without it the OCR runs same-thread (identical output, blocking). pipeline.js is now the wasm-only OCR module with a document lifecycle (startDoc/addPage/finishDoc) plus convertImage. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…r ORT calls) The exact-width grouping runs one recognition call per distinct line width — on a text-dense page (most lines differing) that is nearly one ORT call per line, and every call carries thread-sync overhead under ORT Web. Add width_batches_padded / batch_input_padded: group lines into REC_BATCH chunks sorted by width and pad each to the batch max. The padding is the input buffer's zero value — mid-gray in the model's [-1, 1] space, exactly what PP-OCRv3 trains/serves with in batched inference — so the decode is output-equivalent to one-at-a-time recognition (measured: 0 diffs on a mixed Latin+Cyrillic sample). A 1.5x width-ratio guard bounds padding waste. Wire the two wasm callers (ocr.rs, scanned.rs) to the padded path; the native pipeline keeps the bit-identical exact-width width_batches, so the byte-for-byte conformance baseline is untouched. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…er (fewer ORT calls)" This reverts commit 1c871c7. Padding every line in a batch up to the batch's max width inflates recognition compute (a narrow line padded to a wide one does several times the conv work), and on real pages line widths vary enough that the padding waste outweighs the saved per-call overhead — measured ~9.6-10.4s/page vs ~8.0s/page with the exact-width grouping. With multi-threaded ORT the per-run overhead the batching targeted is already cheap, so the trade doesn't pay. Back to the bit-identical exact-width batches. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
wasm-pack warns when license.workspace resolves but no LICENSE sits next to the crate. Copy the workspace MIT license into the crate directory. Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…ore into tf_core Groundwork for TableFormer in the browser: move the ONNX-free half of the pipeline — the 448 encoder-input preprocessing, the autoregressive loop's structure corrections (xcel→lcel, lcel-after-ucel→fcel) and bbox bookkeeping (tag_H_buf / bboxes_to_merge), span merging, and the OTSL→grid layout — into a new ocr-prep-gated tf_core module. The wasm build will run these exact functions and delegate only the three ONNX graphs to ONNX Runtime Web, the same one-implementation approach that keeps the browser OCR/layout output matching native. Native tableformer.rs (ml feature) now delegates to tf_core: preprocess wraps tf_core::preprocess_input in an ort tensor; predict_otsl uses tf_core::correct; predict_table_structure drives tf_core::BboxBook. Pure code-motion — the owned-value KV-cache fast path and every ort call are untouched, so native output is byte-identical. Unit tests cover the moved pure functions (corrections, bbox bookkeeping, span build, argmax — noting argmax ties to the last index via Rust max_by, kept verbatim). Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Scanned PDFs frequently store pages as JPEG2000 (or JBIG2) images, which pdf.js v6 decodes with a separate wasm module; without a wasmUrl it warns "Ensure that the wasmUrl API parameter is provided" and drops the image, so the rasterized page — and the OCR run on it — comes out blank. Pin pdf.js to 6.1.200 so the module, worker, and asset URLs all match, and pass wasmUrl (plus cMapUrl for CJK and standardFontDataUrl) to getDocument. wasm/ and cmaps/ verified present on the CDN. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…er ocr-prep Both modules are already pure Rust (resample uses only `image`; tf_match only std) but were gated behind `ml`. Re-gate them to ocr-prep and make tf_match a public module so the browser TableFormer path (phase 3) can crop table regions (page→1024px box-average, crop→448 bilinear) and run docling's cell matcher — the same code the native pipeline uses. Native `ml` still pulls them in (ocr-prep ⊂ ml); the ocr-prep-only slice now compiles both. No behavior change. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…tail into tf_core Extract the second half of predict_table_rows — word-to-cell matching (docling's post-processor via tf_match, or the geometric fallback) and the span-expanded rows×cols grid assembly — into tf_core::table_rows, shared with the browser path. The native predict_table_rows keeps its ort head (resample crop → predict_table_structure) and delegates the tail. The parity debug dump (fs) stays native-only (#[cfg(feature = "ml")]) so the shared code compiles for wasm. Pure code-motion; native output unchanged. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…t-web interop Wire the ONNX table-structure model into the browser pipeline. New wasm module tableformer.rs delegates the three graphs (encoder/decoder/bbox) to a stateful JS TfSession and drives the autoregressive loop in Rust on tf_core: preprocess_input → encode → step* (BboxBook bookkeeping, argmax) → bbox → merge_spans → build_table_cells → table_rows. The heavy tensors — the encoder's constant cross K/V and enc_out, and the growing decoder KV-cache — stay JS-side in the session, so each step marshals only the last tag (one int) in and logits+hidden (525 floats) out, avoiding O(steps^2) copies across the boundary (the decoder_kv graph: 6 layers, 8 KV heads, head_dim 64, cross length 784). ScannedConverter gains addPageTf (the lite add_page stays geometric): table regions run predict_table_rows, others stay None, then scanned::assemble_page_with_tables (new — the general assemble the native pipeline uses) lays them out. TableFormer is opt-in so the 225 MB encoder isn't fetched unless a table profile is chosen. Native ml unchanged; wasm target + clippy clean. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Add the browser TableFormer profile to the scan demo. A "TableFormer tables" checkbox opts in; when checked, pipeline.js lazily loads the three graphs same-origin from ./models/tableformer/ (encoder.onnx self-contained ~225 MB; decoder_kv.onnx + bbox.onnx with their .onnx.data sidecars via ort-web's externalData option) and builds JsTfSession — the stateful session the wasm TfSession interop expects: encode stashes the constant cross K/V + enc_out and resets the KV-cache, step runs one decoder round (int64 tag, empty first-step cache [6,1,8,0,64]) and returns logits+hidden, bbox runs the box/class head. Table regions then go through ScannedConverter.addPageTf; the lite geometric path stays the default so the big models aren't fetched unless asked. Models load lazily on the first table conversion with a progress line. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Splitting add_page into a #[wasm_bindgen] wrapper pair plus a plain
add_page_impl left page_count and finish inside the non-wasm_bindgen impl
block, so they stopped being exported to JS ("cur.conv.finish is not a
function"). Close the plain impl right after add_page_impl and put
page_count/finish back in a #[wasm_bindgen] impl block.
Refs docling-project#157
Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Formatting-only: rustfmt reorders the tf_core/tf_match mod declarations and rewraps a couple of tf_core lines. Fixes the pinned fmt·clippy CI job. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Add a cross-origin layout candidate after the local ./models/ ones: a Hugging Face mirror (pivozavrus/docling-rs-models) that sends CORS headers, so the page works when served cross-origin (the hosted phone demo on raw.githack) — GitHub Release assets can't be fetched that way. Local dev still prefers ./models/. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Diagnostic for the browser scanned path: console-log the per-page region labels after refinement, so it's visible whether the layout flagged any `table` region — a scanned table only fills when it does and the region's words are OCR'd (a known gap: prep_region_lines skips non-text labels, so table interiors currently carry no words for the matcher). Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Phones can't open the console, so route the layout region-label histogram to the page instead: the wasm calls a global __docling_diag sink, the worker forwards it to the main thread, and scan.html shows it in a small #diag line under the controls (cleared on each new conversion). The main-thread fallback defines the sink directly. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The browser scanned path never recognized text inside table regions — prep_region_lines skips non-text labels and there's no pdfium text layer — so a detected table had no words for the matcher and rendered empty. Add ocr_prep::prep_table_words: crop each table region, split into lines, then split each line into word tokens by a vertical ink-projection profile (segment_words), and prep each word for recognition with its own page-point box. The browser add_page now OCRs those word crops and appends them to the page cells; assemble routes them into the table region (geometric or TableFormer) instead of stray paragraphs. Word-level boxes are what let the matcher distribute a row across its columns. Native untouched (it reads word boxes from pdfium). Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Mirror the layout-model fallback: try local ./models/tableformer/ first, then the CORS Hugging Face base, so TableFormer works on the hosted (phone) demo. External data is fetched from the same base that served the .onnx. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The models were uploaded to the repo root next to layout_heron_int8.onnx (no subfolder), so point the HF fallback base at MODEL_BASE directly instead of MODEL_BASE + "tableformer/". Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The load fallback swallowed the fetch error and reported a flat "not found", hiding whether it was a 404, an HTTP 503 throttle, or an out-of-memory on the 225 MB encoder. Keep and report the last error. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Add a multi-file picker so the models can be selected from the device instead of fetched: each File is read to an ArrayBuffer, handed to the pipeline (transferred to the worker), and used by name before any network fetch. This avoids the 380 MB TableFormer download on a phone entirely and — since reading a File is a single allocation — sidesteps the double- buffering peak fetchProgress hits on the 225 MB encoder (the likely cause of the earlier load failure). Layout and TableFormer both resolve provided-file → local ./models/ → Hugging Face, in that order. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Rewrite the README's ML section to cover all three stages (OCR, layout, TableFormer docling-project#157) with the model manifest, add a "Run it on your phone" walkthrough (download_dependencies.sh from upstream master, static hosting notes, and the device model-file picker with its device → local → Hugging Face resolution order), and a "Performance & threading" section (coi.js cross-origin isolation for ORT threads, the int8 layout export, and the fp32 encoder as the dominant mobile bottleneck with a pointer to quantize_models.py). Also wrap ocr_lines to satisfy rustfmt. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The earlier note called an int8 TableFormer encoder "the biggest lever." Measuring on a native x86 build over 15 real corpus table crops shows otherwise: the encoder is a ResNet backbone feeding a 6-layer transformer, and the transformer Gemms (not the ~20 convs) are the cost. Conv-only static QDQ keeps output fidelity (enc_out/cross cosine >= 0.995) but is not faster than ORT's fp32 conv kernels; quantizing the attention MatMuls collapses cross-attention fidelity to ~0.85 (garbled structure) for no speed gain. Encoder stays fp32; document the real levers (desktop for heavy tables, multi-table batching) instead of a quantization that does not pay. Refs docling-project#157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
artiz
force-pushed
the
claude/wasm-ml-stage1
branch
from
July 25, 2026 07:54
065e284 to
583b706
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of the browser ML pipeline: scanned images OCR fully client-side.
docling-pdf grows an
ocr-prepfeature (pure Rust + the image crate, no ort/pdfium): the ONNX-free half of the PP-OCRv3 recognition pipeline — line segmentation, crop prep, deterministic width-batching, CTC decode, dictionary handling — moved verbatim from ocr.rs into a public ocr_prep module. ocr.rs now consumes it, so there is exactly ONE implementation of the pre/post-processing; the native path is byte-identical after the refactor (verified: the scanned-tiff conversion diffs empty against the pre-refactor output, full docling-pdf suite green). Any wasm-vs-native drift can therefore only come from the runtime's kernels — the #157 conformance story depends on that property.docling-wasm (default
ocrfeature):ocr_image(bytes, dict, session, to)— decode the image in Rust, segment/prep/batch via ocr_prep, delegate each width-batch to a JSRecSessionwrapper around an ort-web InferenceSession (run(n, h, w, data) -> {data, dims}), CTC- decode, and export a DoclingDocument as Markdown or docling JSON. No layout model yet (stage 2): the whole image is one text region split by the ink-projection profile — right for single-column scans, degraded on complex layouts, and said so in the docs.www/ocr.html: complete drop-an-image demo — ort-web from CDN, the ~10 MB en_PP-OCRv3 model + dictionary fetched from their public hosting and browser-cached, timing readout. Compiles for wasm32-unknown-unknown and the host (rlib tests); clippy clean; the existing wasm CI check is unaffected.
Closes #157
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT